Papers with multi-task benchmark
CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in foundation language models have shown the efficacy of pre-trained models across diverse QA tasks. |
| Approach: | They propose a multi-task benchmark for evaluating causality-aware language models to unify causal QA research. |
| Outcome: | The proposed model outperforms single-task fine-tuned models on the CALM-Bench tasks. |
PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English (2023.acl-short)
Copied to clipboard
| Challenge: | Existing efforts to understand privacy policies are limited by processing the language in a way exclusive to a single task focusing on certain privacy practices. |
| Approach: | They propose a privacy policy language understanding evaluation benchmark to evaluate the understanding of privacy policies across multiple tasks. |
| Outcome: | The proposed framework improves the understanding of privacy policies across multiple tasks. |
GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)
Copied to clipboard
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan
| Challenge: | Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models. |
| Approach: | They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. |
| Outcome: | The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages. |
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)
Copied to clipboard
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
| Challenge: | Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. |
| Approach: | They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities. |
| Outcome: | The proposed model outperforms open-source models but struggles on longer contexts. |
AfroBench: How Good are Large Language Models on African Languages? (2025.findings-acl)
Copied to clipboard
Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, David Ifeoluwa Adelani
| Challenge: | Large-scale multilingual evaluations often include only a handful of African languages due to the scarcity of high-quality data and the limited discoverability of existing datasets. |
| Approach: | They propose a multi-task benchmark to evaluate the performance of LLMs across 64 African languages, 15 tasks and 22 datasets. |
| Outcome: | The proposed benchmark compares LLMs across 64 African languages, 15 tasks and 22 datasets. |
Positive and Risky Message Assessment for Music Products (2024.lrec-main)
Copied to clipboard
| Challenge: | a new approach to content assessment is needed to assess positive and potentially harmful messages in music. |
| Approach: | They propose a multi-task predictive model fortified with ordinality-enforcement to assess positive and potentially harmful messages within music products. |
| Outcome: | The proposed method outperforms task-specific alternatives and can assess multiple aspects simultaneously. |